Audio-driven video portrait synthesis is a crucial and useful technology in virtual human interaction and film-making applications. Recent advancements have focused on improving the image fidelity and lip-synchronization. However, generating accurate emotional expressions is an important aspect of realistic talking-head generation, which has remained underexplored in previous works. We present a novel system in this paper for synthesizing high-fidelity, audio-driven video portraits with accurate emotional expressions. Specifically, we utilize a variational autoencoder (VAE)-based audio-to-motion module to generate facial landmarks. These landmarks are concatenated with emotional embeddings to produce emotional landmarks through our motion-to-emotion module. These emotional landmarks are then used to render realistic emotional talking-head video using a Neural Radiance Fields (NeRF)-based emotion-to-video module. Additionally, we propose a pose sampling method that generates natural idle-state (non-speaking) videos in response to silent audio inputs. Extensive experiments demonstrate that our method obtains more accurate emotion generation with higher fidelity.

Another day, another deepfake: but this time they can sing. New research from Imperial College in London and Samsung's AI research center in the UK shows how a single photo and audio file can be used to generate a singing or talking video portrait. Like previous deepfake programs we've seen, the researchers uses machine learning to generate their output. And although the fakes are far from 100 percent realistic, the results are amazing considering how little data is needed. Getting a bit wackier, why not have everyone's favorite mad monk, Grigori Yefimovich Rasputin, belting out the Beyoncé classic'Halo'?

If you think it's been a problem up to this point, the fight against fake news is about to get a whole lot harder. That is thanks to artificial intelligence technology which is making the creation of so-called "deep fake" videos more convincing at a frankly terrifying rate. The latest development comes from an international team of researchers, lead by Germany's Max Planck Institute for Informatics. They have created a deep-learning A.I. system which is able to edit the facial expression of actors to accurately match dubbed voices. In addition, it can tweak gaze and head poses in videos, and even animate a person's eyes and eyebrows to match up with their mouths -- representing a step forward from previous work in this area.

We present a novel approach that enables photo-realistic re-animation of portrait videos using only an input video. In contrast to existing approaches that are restricted to manipulations of facial expressions only, we are the first to transfer the full 3D head position, head rotation, face expression, eye gaze, and eye blinking from a source actor to a portrait video of a target actor. The core of our approach is a generative neural network with a novel space-time architecture. The network takes as input synthetic renderings of a parametric face model, based on which it predicts photo-realistic video frames for a given target actor. The realism in this rendering-to-video transfer is achieved by careful adversarial training, and as a result, we can create modified target videos that mimic the behavior of the synthetically-created input. In order to enable source-to-target video re-animation, we render a synthetic target video with the reconstructed head animation parameters from a source video, and feed it into the trained network -- thus taking full control of the target. With the ability to freely recombine source and target parameters, we are able to demonstrate a large variety of video rewrite applications without explicitly modeling hair, body or background. For instance, we can reenact the full head using interactive user-controlled editing, and realize high-fidelity visual dubbing. To demonstrate the high quality of our output, we conduct an extensive series of experiments and evaluations, where for instance a user study shows that our video edits are hard to detect.